Method and system for interacting with medical information

ABSTRACT

A system for permitting a medical practitioner to interact with medical information, the system including: a sensing unit for detecting a position of a reference object used to interact with the sensing unit; at least one control unit for determining a gesture performed by the medical practitioner, identifying a command relative to the medical information that corresponds to the gesture, and executing the command in order to display the medical information; generating a graphical user interface including a virtual representation of the reference object and at least one virtual icon and/or a virtual representation of the sensing unit, with each of the at least one virtual icon corresponding to one of a respective mode of operation, a respective user notification, and a respective system setting option; and displaying the GUI along with the medical information.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PCTApplication No. PCT/IB2016/056228 filed Oct. 17, 2016, which claims thebenefit of U.S. Provisional Application No. 62/260,428 filed Nov. 27,2015, the contents of each application hereby incorporated by referencein their entirety.

TECHNICAL FIELD

The present invention relates to the field of methods and systems forinteracting with medical information.

BACKGROUND

There is a desire to provide medical practitioners (e.g. surgeons,interventional radiologists, nurses, medical assistants, other medicaltechnicians and/or the like) with access to the ability to manipulateand/or the ability to otherwise interact with medically relevantinformation such as prior, during and/or after the performance ofmedical, surgical, and interventional procedures and/or operations,and/or the like. Such desired medical information may include, by way ofnon-limiting example, radiological images, angiography images, otherforms of images of the patient's body, other information relevant to apatient undergoing a medical procedure, other information relevant tothe procedure itself, other information related to the condition beingtreated and/or the like. Such desired medical information may beprocured prior to performing the procedure, during performance of theprocedure, and/or after performance of the procedure, and may allowmedical practitioners to formulate or alter their therapeutic planduring image-guided and/or image-dependent medical procedures.

Currently, intra-procedural access to, manipulation of and/orinteraction with medical information such as radiological images takesplace on a computer workstation in a control room located outside of thesurgical sterile environment. Such a computer workstation may access,via suitable network communications or other digital access techniques,information such as archives of image data pertaining to a patient byaccessing picture archiving and communication systems (PACS); digitalimaging and communications in medicine systems (DICOM), hospitalinformation systems (HIS), radiological information systems (RIS) and/orthe like. Such workstations may then display individual images on asuitable display and may permit manipulation of the images via aconventional computer-based user interface—e.g. using a mouse andkeyboard and a software-implemented user interface. Since theworkstation is usually located outside of the surgical sterileenvironment, a medical practitioner such as a radiologist wanting toaccess various images typically has to either: (a) scrub out of aprocedure on one or more occasions during the procedure; or (b) delegatethe task of accessing the desired image(s) to a another person such as atechnologist or a nurse, who then has to operate the workstation underthe direction of the radiologist.

In case (a), the need for the medical practitioner to move back andforth between the non-sterile control room and the sterile surgicalenvironment for purposes of image navigation and interpretation may:increase the risk of contaminating the sterile environment byinadvertently transferring contaminants from the non-sterile controlroom into the sterile environment; extend the time required to completethe surgery, thereby increasing procedural costs; and/or interrupt themedical practitioner's cognitive focus, thereby increasing the medicalrisk for the patient. In case (b), close communication between theradiologists and the technologist operating the workstation is typicallyrequired. Communication of relevant information (e.g. how much to moveor enlarge an image) is difficult and time-consuming and may requireseveral iterations. This process may be made more difficult by the needto use different software platforms, to navigate through vendor-specificmulti-layered menus, and to interact with volumetric images using akeyboard and mouse. In both cases (a) and (b), there are factors thatcontribute to surgeon's fatigue which is a big problem during surgicaland/or interventional procedures.

With an increasing reliance on numerous radiological images forintra-procedural planning and confirmation of targeted therapy, there isa general desire to develop solutions that improve the radiologist'sability to rapidly access, manipulate and/or otherwise interact withlarge amounts of image information (and/or other medically relevantinformation) in an intuitive, comprehensive, and timely manner while inthe sterile environment.

Therefore, there is a need for an improved method and system forinteracting with medical information.

SUMMARY

According to a first broad aspect, there is provided a system forpermitting a medical practitioner to interact with medical information,the system comprising: a sensing unit for detecting at least a positionof a reference object used by the medical practitioner to interact withthe sensing unit; at least one control unit being in communication withthe sensing unit for: determining a gesture performed by the medicalpractitioner using the position of the reference object detected by thesensing unit; identifying a command relative to the medical informationthat corresponds to the received gesture and executing the command inorder to display the medical information on a display unit; generating agraphical user interface (GUI) comprising a virtual representation ofthe reference object and at least one of a virtual representation of thesensing unit and at least one virtual icon, a position of the virtualrepresentation of the reference object within the GUI being chosen as afunction of the position of the reference object detected by the sensingunit, each of the at least one virtual icon corresponding to one of arespective mode of operation, a respective user notification and arespective system setting option; and displaying the GUI on the displayunit along with the medical information.

In one embodiment, the controller is configured for displaying the GUIadjacent to the medical information.

In one embodiment, the controller is configured for displaying the GUIand the medical information on a same display device.

In another embodiment, the controller is configured for displaying theGUI and the medical information on separate display devices beingpositioned adjacent to one another so that the GUI be in a field of viewof the medical practitioner when the medical practitioner looks at thedisplayed medical information.

In one embodiment, the sensing unit is further adapted to detect anorientation of the reference object, an orientation of the virtualrepresentation of the reference object within the GUI being chosen as afunction of the orientation of the reference object detected by thesensing unit.

In one embodiment, the sensing unit comprises a single sensor adapted todetermine the position and the orientation of the reference object anddetermine the gesture performed by the medical practitioner.

In one embodiment, the single sensor comprises an optical sensor.

In one embodiment, the optical sensor comprises a camera.

In one embodiment, the camera comprises one of a 3D camera, a stereocamera and a time-of-flight cameras.

In one embodiment, the camera is configured for imaging a referencesurface.

In one embodiment, the system further comprises a projector forprojecting at least one reference icon on the reference surface imagedby the camera, each one of the at least one reference icon correspondingto a respective one of the at least one virtual icon.

In one embodiment, the reference surface comprises a screen on which atleast one reference icon is displayed, each one of the at least onereference icon corresponding to a respective one of the at least onevirtual icon.

In another embodiment, the sensing unit comprises a first sensor fordetermining the position of the reference object and a second sensor fordetermining the orientation of the reference object, the gesture beingdetermined by one of the first and second sensors.

In one embodiment, the first sensor comprises an electric field sensorfor determining the position of the reference object and the secondsensor comprises an optical sensor for determining an orientation of thereference object.

In one embodiment, the optical sensor comprises a camera.

In one embodiment, the camera comprises one of a 2D camera, a monochromecamera, a stereo camera and a time-of-flight camera.

In one embodiment, the camera is positioned for imaging a region locatedabove the electric field sensor.

In one embodiment, the system comprises a projector for projecting atleast one reference icon on the electric field sensor or around theelectric field sensor, each one of the at least one reference iconcorresponding to a respective one of the at least one virtual icon.

In one embodiment, the system further comprises a screen on which atleast one reference icon is displayed, each one of the at least onereference icon corresponding to a respective one of the at least onevirtual icon and the electric field sensor being positioned on thescreen.

In one embodiment, the reference object comprises a body part of themedical practitioner.

In one embodiment, the body part comprises one of a hand and at leastone finger.

In one embodiment, the reference object is made of one of a conductivematerial and a semi-conductive material.

In one embodiment, the reference object comprises one of a pen, astylus, a ball, a ring, and a scalpel.

In one embodiment, the command corresponds to a given known command froma peripheral device connectable to a computer machine.

In one embodiment, the given known command corresponds to one of a mousecommand, a foot pedal command, a joystick command, and a keyboardcommand.

In one embodiment, the medical information comprises a medical image, a3D model, and any combination or sequence thereof.

In one embodiment, the command relative to the medical informationcomprises a command that causes a change of at least one characteristicof an already displayed medical image.

In one embodiment, the at least one characteristic comprises at leastone of a shape, a size, an orientation, a color, a brightness, text anda contrast.

In one embodiment, the controller is adapted to modify an appearance ofone of the at least one virtual icon upon a given selection by themedical practitioner.

According to another broad aspect, there is provided acomputer-implemented method for allowing a medical practitioner tointeract with medical information, the method comprising: detecting aposition of a reference object used by the medical practitioner tointeract with a sensing unit; determining a gesture performed by themedical practitioner using the detected position of the referenceobject; identifying a command relative to the medical information thatcorresponds to the received gesture and executing the command in orderto display the medical information on a display unit; generating agraphical user interface (GUI) comprising a virtual representation ofthe reference object and at least one of a virtual representation of thesensing unit and at least one virtual icon, the position of the virtualrepresentation of the reference object within the GUI being chosen as afunction of the detected position of the reference object, each of theat least one virtual icon corresponding to one of a respective mode ofoperation, a respective user notification and a respective systemsetting option; and displaying the GUI on the display unit along withthe medical information.

In one embodiment, said displaying the GUI comprises displaying the GUIadjacent to the medical information.

In one embodiment, said displaying the GUI comprises displaying the GUIand the medical information on a same display device.

In another embodiment, said displaying the GUI comprises displaying theGUI and the medical information on separate display devices beingpositioned adjacent to one another so that the GUI be in a field of viewof the medical practitioner when the medical practitioner looks at thedisplayed medical information.

In one embodiment, the method further comprises detecting an orientationof the reference object.

In one embodiment, said detecting the position and the orientation ofthe reference object is performed using a single sensor adapted todetermine the position and the orientation of the reference object anddetermine the gesture performed by the medical practitioner.

In one embodiment, said detecting is performed using an optical sensor.

In one embodiment, said detecting is performed using a camera.

In one embodiment, said detecting is performed using one of a 3D camera,a stereo camera and a time-of-flight cameras.

In one embodiment, the camera is configured for imaging a referencesurface.

In one embodiment, the method further comprises projecting at least onereference icon on the reference surface imaged by the camera, each oneof the at least one reference icon corresponding to a respective one ofthe at least one virtual icon.

In one embodiment, the reference surface comprises a screen on which atleast one reference icon is displayed, each one of the at least onereference icon corresponding to a respective one of the at least onevirtual icon.

In another embodiment, said detecting the position of the referenceobject is performed using a first sensor and said detecting theorientation of the reference object is performed using a second sensor,the gesture being determined using one of the first and second sensors.

In one embodiment, the first sensor comprises an electric field sensorfor determining the position of the reference object and the secondsensor comprises an optical sensor for determining the orientation ofthe reference object.

In one embodiment, the optical sensor comprises a camera.

In one embodiment, the camera comprises one of a 2D camera, a monochromecamera, a stereo camera and a time-of-flight camera.

In one embodiment, the method further comprises positioning the camerafor imaging a region located above the electric field sensor.

In one embodiment, the method further comprises projecting at least onereference icon on the electric field sensor or around the electric fieldsensor, each one of the at least one reference icon corresponding to arespective one of the at least one virtual icon.

In one embodiment, the method further comprises displaying at least onereference icon on a screen, each one of the at least one reference iconcorresponding to a respective one of the at least one virtual icon andthe electric field sensor being positioned on the screen.

In one embodiment, the reference object comprises a body part of themedical practitioner.

In one embodiment, the body part comprises one of a hand and at leastone finger.

In one embodiment, the reference object is made of one of a conductivematerial and a semi-conductive material.

In one embodiment, the reference object comprises one of a pen, astylus, a ball, a ring, and a scalpel.

In one embodiment, the command corresponds to a given known command froma peripheral device connectable to a computer machine.

In one embodiment, the given known command corresponds to one of a mousecommand, a foot pedal command, a joystick command, and a keyboardcommand.

In one embodiment, the medical information comprises a medical image, a3D model, and any combination or sequence thereof.

In one embodiment, the command relative to the medical informationcomprises a command that causes a change of at least one characteristicof an already displayed medical image.

In one embodiment, the at least one characteristic comprises at leastone of a shape, a size, an orientation, a color, a brightness, text anda contrast.

In one embodiment, the method further comprises modifying an appearanceof one of the at least one virtual icon upon a given selection by themedical practitioner.

In the following, a gesture should be understood as a static gesture ora dynamic gesture. A static gesture is defined as a particularconfiguration, position and/or orientation of a hand which substantiallydoes not move during a given period of time. For example, a staticgesture may consist in a closed first with one raised finger. A dynamicgesture is defined as a motion of a hand during a given period of time.The hand may have a particular configuration, position and/ororientation which may be constant or may vary during the motion of thehand. For example, a dynamic gesture may correspond to a rotation of theindex while the other fingers are folded.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the present invention will becomeapparent from the following detailed description, taken in combinationwith the appended drawings, in which:

FIG. 1a is a block diagram of a system for interacting with medicalinformation, in accordance with a first embodiment;

FIG. 1b is a block diagram of a system for interacting with medicalinformation, in accordance with a second embodiment;

FIG. 2 illustrates a system for interacting with medical information,comprising a projector for displaying a user interface on or around anelectric field sensor, in accordance with an embodiment;

FIG. 3 illustrates a system for interacting with medical information,comprising a sensor having displays integrated thereon for displayingicons, in accordance with an embodiment;

FIG. 4 illustrates a system for accessing medical information comprisinga robotic arm, in accordance with an embodiment;

FIG. 5 schematically illustrates an operating room in which the systemof FIG. 3 is installed, and a control room, in accordance with anembodiment;

FIG. 6 illustrates an interaction between a hand and an electric fieldgenerated by an electric field sensor, in accordance with an embodiment;

FIG. 7 illustrates the position of a fingertip relative to an electricfield sensor, in accordance with an embodiment;

FIG. 8 illustrates an air wheel gesture, in accordance with anembodiment;

FIG. 9a illustrates one exemplary menu image;

FIG. 9b illustrates a left swipe gesture performed by a medicalpractitioner, in accordance with an embodiment;

FIG. 9c illustrates an air-wheel gesture performed by a medicalpractitioner, in accordance with an embodiment;

FIG. 9d illustrates a deactivation of an electric field sensor, inaccordance with an embodiment;

FIG. 10a illustrates an electric field sensor secured to a robotic armbeing in a first configuration, in accordance with an embodiment;

FIG. 10b illustrates the robotic arm of FIG. 9a in a second anddifferent configuration, in accordance with an embodiment;

FIG. 11 illustrates an interaction with an electric field sensor beingsensitive to a distance between a hand and its surface, in accordancewith an embodiment;

FIG. 12 illustrates a translation gesture for rotating a 3D image, inaccordance with an embodiment;

FIG. 13 is a flow chart illustrating a method for allowing a medicalpractitioner to interact with medical data, in accordance with anembodiment;

FIG. 14 is a block diagram of a system for allowing a medicalpractitioner to interact with medical information and providing a visualfeedback to the medical practitioner, in accordance with an embodiment;

FIG. 15 illustrates an exemplary graphical user interface to bedisplayed along with medical data, in accordance with an embodiment;

FIG. 16 illustrates the display of a medical image and an overlaygraphical user interface, in accordance with an embodiment;

FIG. 17 illustrates exemplary static gestures, in accordance with anembodiment;

FIG. 18 illustrates an exemplary finger tapping gesture, in accordancewith an embodiment;

FIG. 19 illustrates a system for interacting with medical information,comprising a single display device for displaying thereon both medicalinformation and an overlay GUI, in accordance with an embodiment; and

FIG. 20 illustrates a system for interacting with medical information,comprising two separate display device for displaying medicalinformation and a GUI, in accordance with an embodiment

It will be noted that throughout the appended drawings, like featuresare identified by like reference numerals.

DETAILED DESCRIPTION

The present systems and methods allow a medical practitioner to access,manipulate and/or otherwise interact with medical information via anelectric field sensor (e.g. but not limited to an array of capacitiveproximity sensors). For example, during a medical procedure, the medicalpractitioner may use hand or body gestures (e.g. touchless gestures) tointeract with the electric field sensor in order to control displayedmedical information. The gestures may be based on the configuration,position and/or movement of a practitioner's hand or portion of a handsuch as a finger. For example, the gestures may be based on theconfiguration, position and/or movement of at least one finger of thepractitioner's hand. The gestures may be interpreted based on thelocation/position of the gesture (e.g. the location of thepractitioner's hand, finger or fingertip) relative to the electric fieldsensor. The gestures may additionally or alternatively be based on theconfiguration or movement of the gesture (e.g. the configuration ormovement of the practitioner's hand or finger). Such gesture position,movement or configuration may be relative to the electric field sensor.In another embodiment, the practitioner may hold an object such as anobject made of electrically conductive material in his/her hand in orderto interact with the electric field sensor. For example, the medicalpractitioner may hold a pen, a stylus, a metal scalpel, or the like. Inthis case, the gestures may be based on the configuration, positionand/or movement of the object held by the practitioner. The gestures mayalso be based on the configuration, position and/or movement of theobject held by the practitioner and the configuration, position and/ormovement of the practitioner's hand that holds the object.

Such systems and methods allow the medical practitioner to interact withmedical information or data without the need to scrub out of the sterileenvironment or to leave the bed (in case of a workstation in the cornerof the room) in which the procedure is being performed and without theneed to communicate with technicians located outside of the sterileenvironment. By way of example, medical information or data accessed,manipulated and/or otherwise interacted with during a medical proceduremay include: 2D or 3D medical images such as radiological images,angiography images, or other forms of images of the patient's body,medical videos, 2D or 3D images that are not related to the patient'sbody, information relevant to the patient undergoing the medicalprocedure, information about the procedure itself, and/or the like.Medical information is displayed to the medical practitioner as a resultof the interaction of the medical practitioner with the electric fieldsensor. The displayed information may comprise 2D/3D images, texts,videos, and/or the like.

In one embodiment, the system may comprise a projection device forprojecting a user interface menu image adjacent to the medicalpractitioner in order to provide visual feedback to the medicalpractitioner. For example, the user interface menu may be projectedadjacent to the electric field sensor or around the electric filedsensor.

The electric field sensor is in communication with a controller which isadapted to translate the gesture performed by the medical practitionerand detected by the electric filed sensor into a command relative to themedical data. In one embodiment, the controller is in communication witha display unit or monitor display. The display unit is adapted to renderimages such as medical images, images containing text, graphs, drawings,graphics, etc. The display unit may also be used to display videos. Theexecution of the determined command by the controller causes the displayof medical information on the display unit. For example, a first gestureperformed by the medical practitioner may command the display of a givenmedical image while a second and different gesture may command thedisplay of the medical file of the patient. In another embodiment, thecontroller is in communication with a computer machine that is incommunication with the display unit. In this case, the controller isadapted to transmit the command to the computer machine which executesthe command in order to display an image on the display.

It should be understood that a video is a sequence of images and thatthe expression “displaying an image” may be understood as displaying agiven image of a video or a video. It should also be understood that animage may only comprise text. Similarly, an image may comprise text,pictures, photographs, drawings, tables, graphics, and/or the like.

In an embodiment in which the system is used during a procedure, basedon the interpretation of such gestures, the controller may cause thedisplay unit to render an image (or other information) that is visibleto the medical practitioner. The displayed image may comprise an imageor a portion of an image from a library of images relating to thepatient on whom the procedure is being performed. Based on theinterpretation of such gestures, the controller may manipulate thedisplayed image or display a further image. For example, suchmanipulation may comprise zooming in or out with respect to a particulardisplayed image, panning or otherwise moving a displayed portion of aparticular displayed image; adjusting brightness, contrast and/or colorparameters of a particular displayed image; scrolling through a libraryof images to select a new image for display; and/or the like.

FIG. 1a illustrates one embodiment of a system 10 for allowing a medicalpractitioner to interact with medical information. The system 10comprises at least an electric field sensor or electric field proximitysensor 12 and a controller 14 which is in communication with theelectric field sensor 12 for receiving data therefrom.

The electric field sensor is adapted to generate a predefined electricfield such as a predefined electromagnetic field or a predefinedelectrostatic field, and measure the generated electromagnetic field inorder to detect and identify a gesture. In an embodiment in which theelectric field sensor generates an electric field and when a medicalpractitioner performs a gesture using his/her hand and/or an objectwithin the electric field, the electric field generated by the electricfield sensor is disturbed by the presence of the practitioner's handand/or the object and the electric field sensor detects the variation ofthe electric field by comparing the predefined electric field generatedby the electric field sensor 12 and the electric field measured by theelectric field sensor 12. The variation between the predefined electricfield and the measured electric field corresponds to the distortioncaused by the gesture of the medical practitioner within the electricfield. The electric field sensor 12 is further adapted to determine thegesture that was performed by the medical practitioner from thevariation of electric field, and transmit the determined gesture to thecontroller 14.

It should be understood that any adequate electric field sensor may beused. For example, an electric field sensor comprising an array ofcapacitive proximity sensors may be used. In another example, theelectric field sensor may include an array of electrodes.

In one embodiment, the gesture outputted by the electric field sensor 12corresponds to the 2D or 3D position of the hand of the medicalpractitioner such as the 2D or 3D position of a given point of the handas a function of time. For example, the position of the hand may bedefined as the point of the hand that is the closest to the electricfield sensor 12. For example, the gesture may correspond to the 2D or 3Dposition of a fingertip. In another example, the gesture may correspondto the 2D or 3D position of the fingertip of more than one finger. In afurther example, the gesture may correspond to the 2D or 3D position ofthe tip of an object hold by the medical practitioner. It should beunderstood that a position may also refer to a variation of position.

In another embodiment, the gesture outputted by the electric fieldsensor 12 corresponds to the determined variation of electric field thatoccurs when the medical practitioner performs a gesture within theelectric field generated by the electric field sensor 12. In this case,the controller 14 is adapted to determine the gesture performed by themedical practitioner from the variation of electric field received fromthe electric field sensor 12.

In one embodiment, the gesture outputted by the electric field sensor 12corresponds to a discrete input for the controller 14. In this case, thegesture performed by the medical practitioner is substantially static,i.e. the medical practitioner positions his/her hand, his/her finger,and/or an object at a fixed position within the electric field for agiven period of time. For example, a static gesture may correspond to aposition represented by coordinates (X, Y, Z). In another example, astatic gesture may correspond to a variation of position expressed by(δX, δY, δZ).

In another embodiment, the gesture outputted by the electric fieldsensor 12 corresponds to a continuous input for the controller 14. Inthis case, the gesture performed by the medical practitioner iscontinuous or dynamic, i.e. the medical practitioner substantiallycontinuously moves his/her hand, his/her finger, and or an object withinthe electric field during a given period of time. For example, acontinuous or dynamic gesture may be represented by coordinates asfunction of time (X(t), Y(t), Z(t)).

The controller 14 is adapted to receive the gesture from the electricfield sensor 12 and determine a command or action to be executed. Thecommand to be executed is related to medical information. The controller14 accesses a database in order to determine the command to be executed.The database comprises a set of commands to be executed and each commandis associated with a respective predefined gesture. Each command isrelated to medical information, and more particularly to the display,modification, and/or selection of medical information. Therefore, thecontroller 14 is adapted to retrieve the command to be executed bycomparing the received gesture to the predefined gestures stored in thedatabase. When the received gesture matches a given predefined gesture,the controller 14 identifies the command to be executed as being thecommand that corresponds to the given predefined gesture. For example,the execution of a first command may cause text containing medicalinformation about a patient to be displayed. In another example, theexecution of a second command may cause a medical image to be displayed.In a further example, the execution of a third command may cause therotation of a displayed medical image. In still another example, theexecution of a fourth command may cause a zoom on a medical image.

In an embodiment in which a gesture outputted by the electric fieldsensor 12 corresponds to a discrete position such as coordinates (X, Y,Z) or a continuous position such as coordinates (X(t), Y(t), Z(t)), thedatabase comprises a set of predefined commands to be executed and eachcommand is associated with a respective predefined discrete position ora respective predefined sequence of positions. In this case, thecontroller 14 is configured for comparing the received position to theset of positions stored in the database and identifying the command tobe executed as being the predefined command associated with thepredefined position that matches the received position.

In an embodiment in which the gesture outputted by the electric fieldsensor 12 corresponds to a variation of electric field, the databasecomprises a set of predefined commands to be executed and each commandis associated with a respective predefined variation of electric field.In this case, the controller 14 is adapted to retrieve the command to beexecuted by comparing the received variation of electric field to thepredefined variations electric field stored in the database. When thereceived variation of electric field matches a given predefinedvariation of electric field, the command corresponding to the givenpredefined variation of electric field is identified as being thecommand to be executed. The commands stored in the database are relatedto medical information.

In one embodiment, the database of predefined commands and correspondinggestures is stored locally on the controller 14. In another embodiment,the database of predefined commands is stored externally and thecontroller is in communication with the computer machine on which thedatabase is stored.

As illustrated in FIG. 1a , the controller 14 is in communication with adisplay unit 16. The display unit 16 is adapted to display texts,graphs, images such as medical images, videos thereon. Once the commandto be performed has been identified, the controller is adapted toexecute the command. The execution of the command causes the displayunit 16 to display an image or a portion of an image comprising medicalinformation. As described above, the displayed image may comprise textsuch as information related to the patient. In another example, thedisplayed image may correspond to a medical image. In one embodiment,the controller 14 is in communication with at least one computer machineon which a medical database is stored. The medical database comprisesmedical information such as medical images, medical videos, medicalinformation (e.g. patient files), and/or the like.

In one embodiment, the system 10 further comprises a projector 18 forprojecting a user interface menu image or other useful graphics on asurface. The menu image may comprise at least one icon each representinga different mode of operation for the system 10. In one embodiment, themenu image is projected on and around the electric field sensor 12 sothat the icons be positioned adjacent the electric field sensor 12. Inanother embodiment, the menu image may be projected away from theelectric field sensor 12. In one embodiment, the projector isindependent from the controller 16. In another embodiment, the projectoris in communication with and controlled by the controller 16. In thiscase, the projector may project images representing icons adjacent theelectric field sensor 12. For example, each icon may represent anoperation mode for the system 10 and the controller 16 may be adapted toset the color of a given icon that corresponds to the actual mode ofoperation to a given color in order to provide a visual feedback to themedical practitioner. For example, all icons may be white and when themedical practitioner selects a given operation mode by interacting withthe electric field sensor 12, the controller changes via the projectorthe color of the icon that corresponds to the selected operation mode.For example, the color of the icon corresponding to the selectedoperation mode may be changed to yellow.

While the controller 14 is connected to the display unit 16 and isadapted to execute the command determined according to the detectedgesture, FIG. 1b illustrates one embodiment of a system 20 forinteracting with medical information in which a controller 24 is adaptedto transmit commands to a computer machine 26 that is connected to thedisplay unit 16. In this case, the controller is adapted to receive thegesture from the electric field sensor 12 and determine the command thatcorresponds to the received gesture, as described above with respect tothe controller 14. However, the command is then sent to the computermachine 26 that is adapted to execute the command in order to displaymedical information of the display unit 16. In this case, the controller16 may be seen as an interface between the electric field sensor 12 andthe computer machine 26. The controller 24 is adapted to convert agesture detected by the electric field sensor 12 into a command that isknown and understood by the computer machine. For example, thecontroller 24 may convert gestures detected by the electric field sensor12 into a command that would be generated by a computer peripheral suchas a mouse command (such as a left or right click or a double click) orinto a keyboard command. The computer machine 26 then executes thecommand received from the controller 24 as if the command would havebeen received from a peripheral that is connected to the computermachine 26.

In one embodiment, the system 10 is used during a medical procedure on apatient. In this case, the electric field sensor 12, the controller 14,the display unit 16, and the projector 18, if any, are located in thesterile environment in which the procedure is performed. The controllermay be in communication with the computer machine of the control roomworkstation located in the control room which corresponds to anon-sterile environment. The control room workstation may be incommunication with servers on which medical images and medicalinformation about patients are stored. When a command identified by thecontroller 14 corresponds to displaying a medical image or medical text,the controller 14 sends to the control room workstation a requestindicative of the medical image or the medical information to beretrieved. The control room workstation communicates with the adequateserver to retrieve the information requested by the controller 14. Uponreceiving the requested medical image or medical text, the control roomworkstation transmits the received data to the controller 14 whichlocally stores the received medical image or text in order to display iton the display unit 16.

The same may apply to the system 20 illustrated at FIG. 1b . In thiscase, the computer machine 26 may correspond to the control roomworkstation 26 and the controller 24 is adapted to convert the gesturesreceived from the electric field sensor 12 into commands to betransmitted to the control room workstation 26 that executes thecommands.

In the following, there is described an exemplary system 50 allowing amedical practitioner to interact with medical information during amedical procedure.

As described above, the system may comprise a projector for projecting auser interface menu image. FIG. 2 illustrates one embodiment of a system30 comprising an electric field sensor 31, a controller 32, a computer33, a display unit 34 and a projector 35. The controller 32 is incommunication with the electric field sensor 31 and the projector 35.The projector 35 is adapted to project a user interface menu image whichcomprises four icons 36 a, 36 b, 36 c and 36 d which each represent amode of operation for the system 30. The electric field sensor 31 ispositioned by the medical practitioner on the patient (as illustrated)or adjacent to the bed on which the patient lies.

In one embodiment, the controller 32 is adapted to change the appearanceof the four icons 36 a-36 d in order to provide the medical practitionerwith a visual feedback on the actual operation mode of the system 30, asdescribed below. For example, the controller 32 may change the colorand/or shape of the icon representing the actual mode of operation.

FIG. 3 illustrates one embodiment of a system 37 comprising an electricfield sensor 38, a controller 32, a computer 33, and a display unit 34.The electric field sensor 38 comprises four displays 39 a, 39 b, 39 c,and 39 d integrated on the top surface thereof. The controller 32 isadapted to display icons on each display 39 a-39 d, each iconrepresenting a respective mode of operation for the system.

In one embodiment, the controller 32 is adapted to change the appearanceof the four icons displayed on the displays 39 a-39 d in order toprovide the medical practitioner with a visual feedback on the actualoperation mode of the system 37. For example, the controller 32 maychange the color, brightness, and/or shape of the icon representing theactual mode of operation.

FIG. 4 illustrates a system 50 that comprises an electric field sensor52 adapted to detect hand and body gestures performed by a medicalpractitioner, an articulated robotic arm 54 having at least two degreesof freedom (DOF) (not shown), a visual feedback system (VFS) 56 such asan overhead optical projector for projecting a menu image, an optical 2Dor 3D position tracking device 58 such as a monochrome camera or atime-of-flight camera, for tracking the position of the electric fieldsensor 52, a controller or embedded computer 60, and a display monitor62 for displaying medical information such as medical images.

In one embodiment, the system 50 is used during a medical procedure inan operating room 64 as illustrated in FIG. 5. In one embodiment, themedical practitioner divides his/her workspace on a surgery bed in twosections: a first section 66 where a medical procedure on a patient'sbody is to be conducted, and a second section 68 which corresponds tothe rest of the surgery bed, which is intended for placing variousmedical/surgical tools for quick access. For example, as illustrated inFIG. 4, the lower torso/leg area of the patient on the surgery bed isbeing used for placement of tools, including the electric field sensor52. This location for the electric field sensor 52 enables the surgeonto have easy control over medical images displayed on the monitor 62from the surgery bed. This arrangement also allows the medicalpractitioner not to have to exit the operating room in order to use thecontrol room workstation 70 located in an adjacent non-sterile controlroom 72, and come back into the operating room to continue with themedical procedure.

In one embodiment, the electric field sensor 52 is inserted into asterile bag or container such as a disposable sterile bag so that theelectric field sensor may be used from one surgery to another. Inanother embodiment, the electric field sensor 52 is made disposable andmay be thrown away after a single use during a surgery.

In the illustrated embodiment, the electric field sensor 52 ispositioned on a receiving surface which is the surgery bed in this case.The motorized robotic arm 54 has a first end secured to the ceiling ofthe procedure room and the surgery bed is located within the procedureroom so as to be under the motorized robotic arm 54. The controller 60is secured at the second end of the robotic arm 54. The VFS 56 issecured to the robotic arm 54 adjacent to the controller 60. Thetracking device 58 is secured to the VFS 56.

The controller 60 is in communication with the electric field sensor 52,the robotic arm 54, the VFS 56, and the tracking device 58. In oneembodiment, the system 50 is located in a sterile environment such as anoperating room and the controller 60 is further in communication with aworkstation located in a non-sterile control room which is adjacent tothe sterile room. The workstation may comprise medical informationstored thereon and/or be in communication with at least one server onwhich medical information is stored, such as Picture Archiving andCommunication System (PACS) servers.

It should be understood that any adequate communication methods may beused. For example, wired communication may occur between some of thecomponents of the system 50 while wireless communication, such as Wi-Fi,Bluetooth or Ethernet communication, may occur between other componentsof the system 50.

In the illustrated embodiment, the electric field sensor 52 is aself-contained rectangular pad adapted to generate a pre-calibratedelectric field envelope over its surface for short range 3D sensing.When an object such as a hand is placed above the pad within thegenerated electric field, a distortion occurs in the generated electricfield and part of the generated electric field is shunted to the ground,as illustrated in FIG. 6. The electric field sensor 52 comprises anarray of electrodes that independently measure the disturbance inducedby the object in the generated electric field by detecting the change incapacitance values that are measured individually. The electric fieldsensor 52 is further adapted to determine the 2D or 3D position of theobject that generated the disturbance using the changes in capacitancemeasured by the electrodes. For example, the electric field sensor 52may be adapted to calculate the 3D position of a fingertip P withrespect to an origin O of the sensor's base coordinate frame, asillustrated in FIG. 7. The electric field sensor 52 is further adaptedto transmit in substantially real time the determined gesture, i.e. thedetermined position, to the controller 60.

The controller 60 is adapted to receive the determined gesture from theelectric field sensor 52 and determine a corresponding command to beexecuted. The controller 60 accesses a database containing a set ofpredefined gestures and a respective command for each predefinedgesture. By comparing the received gesture to the set of predefinedgestures stored in the database, the controller 60 identifies the givenpredefined command to be executed. The controller 60 then executes thecommand corresponding to the received gesture and displays the medicalinformation resulting from the executed command on the display monitor62.

In one embodiment, the command to be executed requires medicalinformation stored on the workstation or a PACS server. In this case,the controller 60 communicates with the workstation located in thenon-sterile control room to obtain the medical information. Oncereceived, the controller 60 executes the identified command such asdisplaying medical information received from a PACS server via theworkstation. In this case, the command to be executed may comprise anApplication Programming Interface (API) message

In another embodiment, the command to be executed does not require anycommunication with the workstation located in the non-sterile controlroom. In this case, the controller 60 simply executes the identifiedcommand. In this case, examples of commands may comprise zooming on analready displayed medical image, rotating an already displayed medicalimage, etc. In one embodiment, commands not requiring any communicationwith the workstation located in the control room may correspond to amouse command or a keyboard command that would usually be performed onthe workstation. In such an embodiment, the controller may be providedwith a display integrated therein to display the images.

As described above, the system 50 comprises the VFS 56 which is adaptedto project a menu image on a surface such as on the surgery bed. In theillustrated embodiment, the menu image comprises four icons spaced apartfrom another so that the electric field sensor be positioned between theicons substantially at the center of the menu image. Each iconrepresents a different mode of interaction or operation of thecontroller 60. The controller 60 is further adapted to control the VFS56. For example, the controller 56 may change the color of the icon thatcorresponds to an actual mode of operation to identify the actual modeof operation for the medical practitioner, thereby providing feedback tothe medical practitioner. Examples of operation modes may comprise azoom mode in which a medical practitioner may zoom in or out in animage, a motion mode in which the medical practitioner may move animage, a scroll mode in which the medical practitioner may scroll in amenu, through a series of images, through a sequence of image slices, orthe like, a window level mode in which the medical practitioner mayadjust the brightness and/or the contrast of a displayed image, a panmode allowing the medical practitioner for image panning, an imagechanging mode in which the medical practitioner may switch betweenimages or sets of images, an image reset mode or command fortransforming an image back to its default configuration, an autoplaycommand or mode for starting automatic cycling through a series ofimages or videos in a given sequence, a file editing mode in whichfunctions such as copying, pasting, cutting and the like may beaccessed, an image manipulation mode in which manipulations of imagessuch as merger of at least two images may be performed, a feature markerplacement mode in which placement of markers that correspond to aparticular set of desired features in a set of medical data for easynavigation, etc.

For example, when in the window level mode, a particular position ofhand along x-axis may correspond a particular brightness level and theposition along y-axis affect the contrast level. When in the imagechanging mode and if a practitioner has MRI scans (where each scanconsists of a series of images) for three different patients, a doubleair tap gesture may be used as an image changing command to cyclebetween scans of the three different patients. When in the autoplaycommand, a medical practitioner may animate and cycle through sets ofimages in an MRI scan to better understand the anatomy of the scannedorgan in a quick manner for example.

When in the image manipulation mode, an air tap gesture may be used toselect two images from an X-Ray scan and CT scan for superimposition forexample. Once the selection is completed, a left swipe may impose oneimage over the other so that the medical practitioner may concurrentlyobserve details of both images. When in the feature marker placementmode, a feature may refer to any distinguishing character in the image,such as the position of a certain vein in an image or the position of aparticular image in a series of images. For example, when scrollingthrough a series of images, a medical practitioner may mark images torefer repeatedly by an air tap. Henceforth, he could access the markedimages back and forth by left and right swipes respectively.

In an embodiment in which the system 50 comprises more than one mode ofinteraction or operation, the controller 60 may be adapted to identifytwo types of gestures. The gestures of the first type may be used foractivating a desired mode of operation or passing from one mode ofoperation to another. In an example in which two modes of operationsexist, a single gesture may be used for passing from one mode to theother. For example, when performing the single gesture allows to passfrom the first mode to the second mode. Performing the same gesture asecond time allows passing from the second mode back to the first mode.In another example, a first gesture may be used to activate the firstmode while a second and different gesture may be used for activating thesecond mode. The second type of gestures that may be performed activatescommands once a given mode of operation has been activated. The gesturesof the second type may be different from the gestures of the first type.A same gesture may be used in different modes of operation. However, thecommands activated by the same gesture in the different modes ofoperation will trigger different commands. For example, a given gesturemay allow zooming in in a zoom mode and the same given gesture may allowincreasing the brightness of a displayed image in a brightness mode.Alternatively, the gestures may be unique so that no identical gesturesmay be used in different modes of operation.

As described above, the system 50 further comprises a position trackingdevice 58 which is in communication with the controller 60. The positiontracking system 58 is adapted to detect the presence of an object anddetermine the position of the object. The position tracking system isfurther adapted to transmit the position of the object to the controller60 which is adapted to control the position and configuration of thearticulated robotic arm 54. In one embodiment the object tracked by theposition tracking system is the electric field sensor 52. In this case,the controller 60 is adapted to ensure that the icons projected by theVFS 56 remain positioned around the electric field sensor 52 when theposition of the electric field sensor 52 is changed. In this case, thecontroller 60 may be adapted to use the received position of theelectric field sensor 52, determine an adequate position for the VFS 56for ensuring that the icons be positioned around the electric fieldsensor located at the new position, determine the configuration of therobotic arm 54 in order to position the VFS at the adequate position,and modify the configuration of the robotic arm 54 according to thedetermined configuration.

In one embodiment, the system 50 further comprises a speaker in order toprovide an audio feedback to the medical practitioner. In this case, theVFS 56 may or may not be omitted.

In one embodiment, the VFS 56 may be replaced by a display adapted todisplay icons representative of the possible operation modes. In thiscase, the electric field sensor 52 may be positioned or secured to thedisplay so that the icons displayed on the display be located on oraround the electric field sensor 52. The controller 60 is then adaptedto control the display. For example, the controller may change theappearance of the icon that corresponds to the actual mode ofinteraction, such as the color and/or shape of the icon, therebyproviding feedback to the medical practitioner. In a further embodiment,a receiving surface having icons printed thereon may be used to help themedical practitioner. In this case, the electric field sensor 52 may besecured to the receiving surface or simply positioned thereon. The iconsare located on the receiving surface so as to position around theelectric field sensor 52. For example, the receiving surface may be asubstantially rigid plate, a piece of fabric to be deposited on thesurgery bed, etc.

In a further embodiment, the VFS 56 may be omitted and the controllermay display the menu icons representative of the different modes ofoperation directly on the display unit 62.

In the following, there is presented some exemplary gestures that may beused to have commands executed. A first exemplary gesture may correspondto a swipe gesture. Performing a sweeping motion using a fingertip or ahand from one edge of the electric field sensor 52 to an opposite edgemay be associated with a given command. In one embodiment, four swipegestures may be recognized by the controller 60 and each associated witha respective command: swipe from left to right, swipe from right toleft, swipe from top to bottom, and swipe from bottom to top. Forexample, swiping from top to bottom may be associated with passing fromone first mode of interaction to a second mode of interaction whileswiping from bottom to top may be associated with passing from thesecond mode of operation back to the first mode of operation. In thesame or another example, swiping from left to right may be associatedwith passing from a first medical image to a second medical image whileswiping from right to left may be associated with passing from thesecond medical image back to the first medical image.

A second exemplary gesture corresponds to an air-wheel gesture asillustrated in FIG. 8. Using a fingertip or a hand, the medicalpractitioner performs a circular motion in a plane substantiallyparallel to the surface of the electric field sensor 52. This circularmotion provides a counter which is increased or decreased according toits motion direction, i.e. clockwise or counter-clockwise. For example,an air-wheel gesture may be used by the medical practitioner to scrollin a drop-down menu to select an image to be displayed.

A third exemplary gesture may correspond to an air tap gesture. An airtap gesture is performed by having the medical practitioner bringinghis/her fingertip down towards the electric field sensor 52 and thenbringing it back up quickly. The medical practitioner may or may nottouch the electric field sensor 52 while executing an air tap gesture.An air tap gesture may be associated with a left click command of amouse for example.

Another exemplary gesture may correspond to a double air tap. A doubleair tap is performed by having the medical practitioner executing twoair taps successively in a short period of time. For example, a doubleair tap may be associated with the same command as that associated witha double left click of a mouse. While an air tap refers to a touchlessuse of the electric field sensor 52, it should be understood that agesture may include touching the surface of the electric field sensor52. For example, touching the surface of the electric field sensor 52may correspond to a mouse click.

While in the illustrated embodiment, it is secured to the robotic arm54, it should be understood that the controller 60 may be positioned atany other adequate location as long as it remains in communication withat least the electric field sensor 52. Similarly, the electric fieldsensor 52 may be positioned at any adequate location within theoperating room.

FIG. 9a illustrates one exemplary menu image 80 that may be displayed bya projector such as the VFS 56. The menu image 80 comprises four icons82-88 which each corresponds to a respective mode of interaction. Themenu image 80 is substantially centered on an electric field sensor 90so that the icons 82-88 be positioned around the electric field sensor90. As a result, the icon 82 is located on top of the electric fieldsensor 90, the icon 84 is located on the right of the electric fieldsensor 90, the icon 86 is located below the electric field sensor 90,and the icon 88 is located on the left of the electric field sensor 90.

In one embodiment, a given mode of interaction is activated byperforming a swipe gesture in direction of the icon 82-88 correspondingto the desired mode of interaction. For example, a medical practitionermay desire to zoom in a displayed medical image and the icon 88 may beassociated with the mode of interaction allowing the medicalpractitioner to zoom in the displayed medical image. In this case, theicon 88 is referred to as a zoom icon. FIGS. 9b and 9c illustrates amethod for activating the zoom mode by performing a swipe in thedirection of the icon corresponding to the desired mode of interaction.In the illustrated example, the medical practitioner performs a swipegesture in order to activate the zoom mode of operation represented bythe zoom icon 88. In FIG. 9b , the medical practitioner positionshis/her hand on top of the electric field sensor 90 adjacent to theright end of the electric filed sensor 90 and performs a left swipegesture by moving his/her hand towards the left, i.e. towards the zoomicon 88 corresponding to the desired mode of interaction. The left swipegesture is detected by the electric field sensor 90 and a signalindicative of the determined gesture is sent by the electric fieldsensor 90 to the controller such as controller 60. The controller thendetermines that the operation mode of interaction that corresponds tothe left swipe gesture and activates the corresponding mode ofinteraction.

In another example, the medical practitioner may activate the panninginteraction mode by performing an up swipe gesture, i.e. a swipe gesturefrom bottom to top towards icon 82. Once in the panning interactionmode, changing the position of the practitioner's hand in a 2D planeabove the electric field sensor 90 results in image panning.

In one embodiment, upon activation of a given mode of interaction, thecontroller may modify the menu image by modifying the appearance of theicon corresponding to the activated mode of interaction, as describedabove.

In the same or another embodiment, the controller may modify thedisplayed menu image by adding and/or removing displayed icons. Forexample and as illustrated in FIG. 9a , the controller may add two icons92 and 94 in the projected menu image 80 upon activation of the zoominteraction mode following the left swipe gesture of the medicalpractitioner. In the illustrated embodiment, the icons 92 and 94 arepositioned within the menu image 80 so as to be projected on theelectric field sensor 90. The icons 92 and 94 are designed to guide themedical practitioner to interact with the controller while in the zoominteraction mode. The icon 92 represents a clockwise oriented arrow inwhich a “+” sign is inserted, which indicates to the medicalpractitioner that a zoom-in may be done in a displayed image performinga clockwise air-wheel gesture. The icon 94 represents an anticlockwiseoriented arrow in which a “−” sign is inserted, which indicates to themedical practitioner that a zoom-out may be done in a displayed imageperforming an anticlockwise air-wheel gesture. In order to perform anair-wheel, the medical practitioner points a fingertip towards theelectric field sensor 90, as illustrated in FIG. 9c , and moves his/herfingertip to perform a circular or semicircular movement.

Once he has performed his/her desired action, the medical practitionermoves his/her hand away from the electric field sensor 90 and after apredefined period of time, the electric field sensor 90 deactivates, asillustrated in FIG. 9d . Once deactivated, the electric field sensor 90ignores any movement of object that it may detect until reactivation.For example, the electric field sensor 90 may be reactivated by holdingan object such as a hand above its surface at a substantially constantposition for predefined period of time. It should be understood that anyadequate gesture may be used for activating or deactivating the electricfield sensor 90.

In an embodiment in which a robotic arm is used to control the positionof a projector relative to that of the electric field sensor in order toproject a menu image on and around the electric field sensor, therobotic arm allows maintaining the projected menu image on the electricfield sensor when the position of the electric field sensor is changed.In addition to ensuring that the menu image will substantially always beprojected on the electric field sensor, this further ensures that theprojector will substantially always be located above the electric fieldsensor without obstructing the view of the medical practitioner who willalways be allowed to see the displayed image.

In an embodiment in which a robotic arm is present, the electric fieldsensor may be mounted on the robotic arm as illustrated in FIG. 10a . Inthis case, a display positioned adjacent to the electric field sensormay surround the electric field sensor in order to display menu iconsaround the electric field sensor. Alternatively, a display may bepositioned adjacent to the electric field sensor in order to displaymenu icons. In such an embodiment, a position tracking device such asdevice 58 may be present in order to track the position of the medicalpractitioner such as the position of a hand of the medical practitionerand the controller may be adapted to control the configuration of therobotic arm in order to position the electric field sensor at a givendistance from the medical practitioner or the hand of the medicalpractitioner, as illustrated in FIG. 10b . In this case, the electricfield sensor may always be easily accessible for the medicalpractitioner.

While the above description refers to a motorized robotic arm, it shouldbe understood that another arm or structure may be utilized to supportthe projector, the electric field sensor, the position tracking device,and/or the like. For example, a passive articulated arm secured to theceiling of the operation room may be used. In this case, theconfiguration of the arm may be changed manually by a medical operator.The structure may even be a rolling floor table on which the projector,the electric field sensor, the position tracking device, and/or the likemay be positioned.

In an embodiment in which commands are associated with air-wheelgestures, the diameter of the circle or semicircle performed during anair-wheel gesture may influence the command associated with theair-wheel gesture. For example, an air-wheel gesture having a firstdiameter may be associated with a first action to be executed while thesame air-wheel gesture having a second and different diameter may beassociated with a second and different action to be executed. In anotherembodiment, performing an air-wheel gestures with different diameter maytrigger a same action to be executed but a characteristic of the actionis dependent on the diameter of the air-wheel gesture. For example, if azoom activity or a scroll activity is associated with an air-wheelgesture, the diameter of the air-wheel gesture may vary the sensitivityor the speed of the activity. For example, if a full turn of fingerduring an air-wheel gesture results in scrolling past 10 images,increasing the diameter of the air-wheel gesture would make a full turnof finger scroll past 20 images. Such a feature provides the system withadditional precision and resolution in the actions to be executed.

In one embodiment, the electric field sensor is adapted to determine thedistance between the object hold by the medical practitioner or the handof the practitioner and the determined distance is transmitted to thecontroller along with the identified gesture. In this case, thedetermined distance may influence the action to be performed. In oneembodiment, a given gesture performed at a first distance from theelectric field sensor may be associated with a first action to beexecuted, such as zooming, while the same gesture performed at a secondand different distance from the electric field may be associated with adifferent action to be executed, such as panning. In another embodiment,performing a given gesture at different distances from the electricfield sensor may be associated with a same action to be executed but acharacteristic of the action may depend on the distance between the handof the medical practitioner and the electric field sensor, asillustrated in FIG. 11. In one embodiment, the electric field sensordetermines the distance between its top surface and the hand of thepractitioner or an object held by the practitioner while performing thegesture, and the determined distance may be used to vary the speed atwhich the action corresponding to the executed gesture. For example, thecloser the hand of the medical practitioner is from the electric fieldsensor, the lower the speed of the corresponding action may be. Forexample, when the medical practitioner has selected the zooming mode ofinteraction, the medical practitioner may perform an air-wheel gesturein a plane substantially parallel to the surface of the electric fieldsensor in order to zoom in or out in a displayed image. If the air-wheelgesture is performed in proximity of the surface of the electric fieldsensor, the speed of the zooming may be less than the speed of thezooming resulting from an air-wheel gesture performed farther away fromthe surface of the electric field sensor. The same or reverse may applyfor other actions to be executed such as panning.

In one embodiment, the electric field sensor may be adapted to detect toat least two different gestures performed substantially concurrently.For example, the electric field sensor may be adapted to detectconcurrent translations and rotations of the object hold by the medicalpractitioner or the hand of the practitioner in order to detect combinedgestures. For example, rotating the hand according to a rotation axisparallel to the surface of the electric field sensor may trigger therotation of a displayed image while translating or swiping the hand in aplane substantially parallel to the surface of the electric maytranslate the displayed image. If the hand is concurrently translatedand rotated, the displayed image is also concurrently rotated andtranslated.

In one embodiment, a translation gesture may be interpreted as a commandfor rotating a displayed image. As illustrated in FIG. 12, translating ahand along a given axis of the electric field sensor may be convertedinto a rotation of a displayed 3D image about a given axis. For example,translating a hand 5 cm in the x direction and 10 cm in the y directionwould be interpreted by the controller as a 10 degrees rotation of the3D image about the Ix axis and a 20 degrees rotation of the 3D imageabout the Iy axis, respectively.

In one embodiment, intuitive inertia may be added to 3D gestures. Forexample, an inertia effect may be added to air-wheel gesture control inorder to scroll through large series of images or zooming in/out withouttoo much physical effort. In one embodiment, three distinct modes mayexist: 1) slow mode 2) fast mode 3) inertial mode.

The slow mode is activated when the speed of air-wheel gesture is belowa given threshold. In this mode, raw air-wheel input from practitioneris directly translated to scroll or zoom commands for accurate 1 to 1control of the image set. If the air-wheel is executed at a speed abovethe given threshold, the fast mode is activated. Slow scroll enablespractitioner to navigate an image-set frame by frame.

In the fast mode, as long as the medical practitioner executes anair-wheel gesture at a speed that is greater than the given threshold,multiple image-set scrolling or faster zooming occurs. However, once themedical practitioner stops the air-wheel gesture, the scrolling orzooming action does not stop, unlike in the slow scroll mode. Insteadthe inertial mode is activated.

Before the inertial mode be activated, the latest speed at which themedical practitioner executed the air-wheel gesture is recorded. Therecorded speed is then used as an input to calculate the “kick”, orinitial velocity, that the automatic air-wheel will receive for inertialscrolling. Once the inertial mode is activated, the system is made tocontinue zooming/scrolling even when no input from user is received. Anelastic rubber band effect is emulated during inertial mode for smoothexperience, where automatic air-wheel is fast initially, and deceleratesslowly to a stop over a predefined period of time.

In one embodiment and as described above, the system may comprise aspeaker controlled by the controller for providing an audio feedback tothe medical practitioner who operates the electric field sensor. Theaudio feedback may provide the medical practitioner with an easyreference to navigate through various menus. When a particular gestureis successfully recognized by the controller, a unique and respectiveaudio cue (e.g. a beep or series of beeps) intimates the practitioner ofthis change in system state. This may allow the medical practitioner touse the electric field sensor without ever needing to take their eyesoff the display monitors.

In one embodiment, the system may comprise a microphone connected to thecontroller and a medical practitioner may input voice commands via themicrophone in addition to the commands generated via the gesturesdetected by the electric field sensor. For example, the medicalpractitioner may say “scroll” in order to activate the scroll mode andthen use and air-wheel gesture to scroll through images or the like.

FIG. 13 illustrates one embodiment of a method 150 for allowing amedical practitioner to interact with medical data such as a medicalimage. It should be understood that the method 150 is implemented usingthe system 10, 50.

At step 152, the controller is in stand-by and waits for the detectionof a gesture by the electric field sensor. At step 154, the controllerdetermines whether a gesture has been detected by the electric fieldgesture. If a gesture is detected, the controller determines whether acalibration is required at step 156. In one embodiment, the electricfield sensor receives both high and low frequency signals fromelectrodes. The high frequency signals usually correspond to electricalnoise in the system. It is determined that a calibration is requiredwhen noise to low frequency signal ratio is greater than a predefinedthreshold. If no gesture is detected, the duration of the period duringwhich no gesture has been detected is compared to a time durationthreshold at step 158. If the duration during which no gesture has beendetected is equal to or less than the time duration threshold, then themethod returns to step 152. If the duration during which no gesture hasbeen detected is greater than the time duration threshold, step 156 isexecuted.

If the controller determines that no calibration is required, thecontroller projects a menu image on the electric field sensor and startsa mouse emulation on a remote workstation at step 160, i.e. thecontroller starts sending commands that correspond to mouse commandsupon detection of corresponding gestures. If the controller determinesthat a calibration is required, a calibration is performed using currentnoise to signal ratio normalization.

At step 164, the positon of the electric field sensor is tracked using aposition tracking device. Optionally, the projection surface may also bescanned by the position tracking device or any other adequate devicesuch as an IR Depth camera, a stereo camera system, or the like. Thescan of the projection image allows determining any irregularities onthe projection surface such as portions of the projection surface thatare not orthogonal to the projector axis.

At step 166, the controller manipulates the robotic arm in order toposition the projector above the electric field sensor. At step 168, thecontroller pre-distorts the image to be displayed for perspectivecorrection. By pre-distorting the image, it is possible to modify theimage projected by the projector so that the projected image appearsnormal even if the projection surface presents irregularities.

When it determines that a gesture has been detected by the electricfield sensor at step 170, the controller commands the projector toproject the menu image on the electric field sensor. At step 172, thecontroller commands the projector to highlight the menu icon thatcorresponds to the mode of interaction selected by the medicalpractitioner by performing the detected gesture. At step 174, thecontroller executes the command associated with the detected gesturesuch as a given manipulation of a displayed medical image. Then themethod returns to step 154.

In one embodiment, a first set of some gestures is to be used in orderto select a given mode of operation such as the zoom mode, the pan mode,etc. Once the given mode of operation has been selected, a second set oficons gestures may be used to interact with the medical information. Inone embodiment, the first and second sets of gestures are different sothat no gesture present in the first set can be contained in the secondset. In another embodiment, a same gesture may be contained in both thefirst and second sets of gestures. In this case, a given gesture may beused to activate a given mode of operation, and once the given mode ofoperation has been activated, the same gesture may be used to perform adifferent action, such as zooming in, for example.

In one embodiment, the projector may be adapted to display a first setof menu icons each corresponding to a given mode of operation of thesystem. Once a given mode of operation has been activated using acorresponding gesture, the projector may be adapted to project a secondset of menu icons each corresponding to a respective interaction withthe medical data within the selected mode of operation. In oneembodiment, once the given mode of operation has been activated, thefirst set of icons may no longer be projected and only the second set oficons is projected. In another embodiment, both sets of icons areconcurrently displayed once the given mode of operation has beenactivated

The following provides an exemplary vocabulary for gestures based onpalm-down horizontal hand motions and single-finger movement:

Fist/Finger: Moves the mouse cursor on the screen.

Single Air Tap: Region and system state sensitive.

Double Air Tap: Reset image

Left Swipe: Activate zoom mode

Right Swipe: Activate scroll mode

Up Swipe: Activate pan mode. Pan by moving cursor to a desired locationusing one finger.

Down Swipe: Activate window level mode. The display surface is dividedinto four quadrants for example, each representing a pre-set windowlevel. Hovering above a particular quadrant selects a desired imagebrightness and contrast.

It should be understood that combinations of the above-presentedexemplary gestures and/or variations of these exemplary gestures may beused.

In one embodiment, the above described system and method for interactingwith medical data allows for intuitive and ergonomic touchlessinteractions with medical images in a sterile environment, with specificattention to real-life requirements and constraints of asurgical/interventional environment.

In one embodiment, using electric field sensing requires substantiallyless processing power and operates independently of its surrounding'slighting conditions. These features ensure that an object of interestmay be tracked robustly with little risk of failures and errors. Suchreliable tracking of a practitioner's hand in real-time also allows fordetection and implementation of several novel ergonomic gestures, forexample:

In one embodiment, the present system allows the medical practitioner tointeract with the medical information independently of the lightingconditions in which the system is used. Since, the operation of theelectric field sensor is independent on the lighting conditions, it ispossible to interact with the medical information even in the dark, forexample.

In one embodiment, the present system and method make it easy for anovice user to quickly pick up and get comfortable with the system. Thenatural and easy to remember gestures ensure a very small learning curvefor a new user. Within just a few minutes of using the present system, apractitioner can get comfortable enough to use it without breakinghis/her line of sight with monitors on which medical images aredisplayed.

In one embodiment, the electric field sensor is used in a non-contactmanner or touchlessly. This allows reducing any risk of transferring anycontaminants that could be present on the surface of the electric fieldsensor.

In one embodiment, the electric field sensor may be operated even whenits field of view is obstructed by an object. For example, the electricfield sensor is operable even when it is covered by a surgery drape.

While in the above description an electric field sensor is used fordetecting a gesture performed by a medical practitioner and onlydisplaying medical information according to the gesture, the followingpresents a system for further providing the medical practitioner with avisual feedback on its interaction with a sensor.

FIG. 14 illustrates one embodiment of a system 200 for interacting withmedical data, such as medical images, and providing a user, such as amedical practitioner, with a visual feedback on its interaction with thesystem 200. The system 200 comprises at least a sensing unit 202, acontroller 204 and a display unit 206. The sensing unit 202 is adaptedto detect a gesture performed by the medical practitioner who uses thesystem 200. As described above, the medical practitioner may use one ofhis hands or an object to interact with the sensing unit 202. Thesensing unit 202 is further adapted to detect the position andorientation of the hand or the object used by the medical practitioner.

The controller 204 is adapted to execute a command according to thegesture detected by the sensing unit 202 and display medical informationon the display unit 206, as described above. The controller 204 isfurther adapted to generate a graphical user interface (GUI) and displaythe generated GUI on the display unit 206 such as on the same screen onwhich the medical data is displayed. The GUI provides the medicalpractitioner with a visual feedback on its interaction with the sensingunit 202.

In one embodiment, the GUI comprises at least one icon eachcorresponding to a respective command to be executed upon the detectionof a respective gesture. The GUI further comprises a graphical orvirtual object for representing the hand or the object used by themedical practitioner to interact with the sensing unit 202. For example,the graphical object may correspond to an arrow. The position of thegraphical object relative to the icons in the GUI is chosen as afunction of the position of the hand or object used by the medicalpractitioner. In one embodiment, the position of the graphical objectrelative to the icons in the GUI is chosen as a function of the 3Dposition of the hand or object used by the medical practitioner. In oneembodiment, the position of the graphical object relative to the iconsin the GUI is chosen as a function of the position of the hand or objectused by the medical practitioner relative to the sensing unit 202.

It should be understood that determining the position of the hand or theobject may correspond to determining the position of a single point ofthe hand or the object. For example, the position of the hand may bedetermined by knowing the position of a fingertip. In another example,the position of a pen may be determined knowing the position of the endof the pen opposite to the hand holding the pen.

The orientation of the graphical object relative to the icons in the GUIis also chosen as a function of the orientation of the hand or objectused by the medical practitioner relative to the sensing unit 202.

In one embodiment, when the surgeon is busy with the delicate work ofsurgery, interacting with medical images should not interrupt thisdelicate surgical workflow. Therefore, the surgeon cannot keep lookingat the position of his hand relative to the sensing unit 202 and goback/forth between his hand and the display unit 206 on which themedical data is displayed. When the GUI provides the surgeon with avisual feedback of the relative position between his hand and thesensing unit and the orientation of his hand, the surgeon's gaze remainson the display and he does not need to look at his hand or the sensingunit 202 for interacting with the system 200.

In one embodiment, the position and orientation of the object used bythe medical practitioner are monitored substantially continuously sothat the position and orientation of the graphical object be updated inthe GUI in substantially real time.

In one embodiment, the GUI further comprises a graphical or virtualrepresentation of the sensing unit 202, hereinafter referred to as thegraphical sensing unit. In this case, the icons are each positioned at arespective location relative to the graphical sensing unit within theGUI. For example, the icons may be positioned over the graphical sensingunit or around the graphical sensing unit. FIG. 15 illustrates anexemplary GUI 230 which comprises four icons 232, 234, 236 and 238 whicheach correspond to a respective mode of operation, a graphical sensingunit 240 and a virtual hand 242 for presenting the hand of the medicalpractitioner. The position of the icons 232-238 and that of thegraphical sensing unit are fixed within the GUI 230 while the positionand orientation of the virtual hand 242 is adjusted within the GUI as afunction of the hand position and orientation of the hand of the medicalpractitioner. This allows the medical practitioner to know the positionof his hand relative to the sensing unit 202 while only looking at thedisplay unit 206 where the medical data is displayed without needing tolook at his hand or the sensing unit 202.

In one embodiment, the controller 204 uses object recognition todetermine the object used by the medical practitioner to interact withthe sensing unit 202. In this case, the graphical object representingthe object used by the medical practitioner may be a graphicalrepresentation of this object. For example, if the medical practitioneruses one of his fingers to interact with the sensing unit 202, thecontroller 204 may generate a graphical representation of a closed firstwith a finger sticking out to represent the hand of the medicalpractitioner. If the medical practitioner uses a pen to interact withthe sensing unit 202, the controller 204 may generate a graphicalrepresentation of a pen and display this graphical representation withinthe GUI. In one embodiment, the system 200 comprises a databasecontaining predefined virtual objects and the controller 204 is adaptedto select a given one of the predefined virtual objects according to theperformed object recognition. The controller 204 may the virtual objectof which the shape matches that of the real object.

In one embodiment, the GUI is displayed adjacent to the medical data onthe same display unit. In this case, the screen of the display unit 206may be divided into two sections, i.e. a first section for displayingthe medical data such as a medical image, and a second section fordisplaying the GUI. In another embodiment, the GUI corresponds to anoverlay GUI which is displayed over the displayed medical data asillustrated in FIG. 16.

In one embodiment, the sensing unit 202 comprises a single sensor todetermine both the gestures performed by the medical practitioner andthe position and orientation in space of the object used by the medicalpractitioner for interacting with the medical data. For example, thesingle sensor may be an optical sensor such as a camera. In anotherembodiment, the single sensor may comprise an ultrasonic sensor arraycombined with wearable inertial measurement units (IMU's) for gesturerecognition and determination of the position and orientation of thehand or object. In this case, gestures performed by the medicalpractitioner are determined by the controller 206 using the dataacquired by the sensor such as images acquired by a camera. Thecontroller 204 then determines the commands corresponding to thegestures and display medical data on the display unit 206 according tothe commands. The controller 204 is further adapted to display a virtualrepresentation of the object used for interacting with the system 200within the GUI displayed on the display unit 206. In this embodiment, acamera such as a 3D camera, a stereo camera system comprising at leasttwo cameras, a time-of-flight camera or the like may be used. It shouldbe understood that any adequate optical sensor adapted to receive anddetect light from the environment and interpret the detected light into2D or 3D information to allow detection of a gesture and the positionand orientation of a hand of object may be used.

In one embodiment and as described above, icons are displayed on areference surface that is imaged by the camera. For example, a projectormay display the icons on the reference surface as described above. Inanother embodiment, the reference surface may comprise a screen on whichthe icons are displayed. The displayed icons each correspond to arespective icon contained in the GUI. In one embodiment, the relativeposition between the icons contained in the GUI corresponds to therelative position between the icons that are displayed on the referencesurface.

In another embodiment, the sensing unit 202 uses the fusion between twodifferent sensors for determining the gestures performed by the medicalpractitioner and the position and orientation of the hand or object usedby the medical practitioner. A first sensor is used for detecting theposition of the hand or object used by the medical practitioner whilethe second sensor is used for the detection of the orientation of thehand or object. For example, the sensing unit 202 may comprise anelectric field sensor for detecting the position and an optical sensorsuch as a camera is used for imaging the hand or object in order todetermine its orientation. It should be understood that any adequatemethod for determining the orientation of the hand or object from theimages taken by the optical sensor may be used. A camera such as such asa 2D camera, a monochrome camera, a stereo camera, a time-of-flightcamera or the like may be used. The gestures can be detected by thefirst and/or second sensor. In one example, the position of the hand orobject used by the medical practitioner for interacting with the sensingunit 202 is determined from the data acquired by an electric fieldsensor which measures the position of the fingertip or the end of theobject held by the medical practitioner while the orientation of theobject is determined using the images acquired by a camera. The gesturesmay be determined using the electric field sensor and/or the camera. Forexample, static gestures such as the gestures illustrated in FIG. 17 maybe determined using the images acquired by the camera while at leastsome dynamic gestures may be determined using the data acquired by theelectric filed sensor. It should be understood that at least somedynamic gestures may also be determined using the images acquired by thecamera.

FIG. 18 illustrates one embodiment of a dynamic gesture which may bedetected by an adequate camera. The illustrated dynamic gesturecorresponds to a finger tapping. In order to perform the finger tapping,the medical practitioner extends his index finger and moves the tip upand down to simulate a tap on a button. The medical practitioner mayalso extend his thumb and tap his index finger on its thumb in order to‘feel’ like a button. Such as dynamic gesture may be interpreted by thecontroller 204 as a mouse click command.

It should be understood that, when a camera is used in connection withan electric field sensor, the camera is positioned so as to image theelectric field sensor or a region above the electric field sensor inwhich the reference object, e.g. the hand of the medical practitioner orthe object used by the medical practitioner, is present.

In one embodiment, the size and/or position of the GUI displayed on thedisplay unit 206 is adjustable. For example, using adequate gestures,the medical practitioner may input a command to move the GUI to anotherlocation within the screen, increase or decrease the size of the GUI,and/or suppress the display of the GUI.

While the above description refers to a display unit comprising a singlescreen on which both medical information/data and a GUI are displayed,the person skilled in the art will understand that the display unit maycomprise more than one screen. For example, the display unit maycomprise a first screen on which the medical information such as amedical image is displayed and a second and separate screen on which theGUI is displayed. In this case, the relative position between the twoscreens and the dimension of the screens are chosen so that the GUIdisplayed on the second screen be in the field of view of the medicalpractitioner while he is looking at medical information displayed on thefirst screen of the display unit. This may be achieved by having thesecond by positioning the second screen adjacent to the first screen sothat the second screen be within the field of view of the medicalpractitioner while he is looking at the first screen. For example, thesecond screen may in physical contact with the first screen andpositioned below the first screen. The second screen may be chosen to besmaller than the first screen.

It should be understood that the first and second screens may be part ofa single display devices. Alternatively, the first and second screensmay each be part a respective display devices so that the display unitcomprises two separate display devices.

FIG. 19 illustrates an exemplary system 300 adapted to display a GUI anda medical image on the same screen. The system 300 comprises an electricfield sensor 302, a camera 304, a projector 306 for projecting icons, acontroller 310, a computer machine 312 and a single display device 314.The controller 310 is in communication with the electric field sensor302, the camera 304, the projector 306 and the computer machine 312.

The controller 310 is adapted to receive images taken from the camera304 and determine at least the orientation of the hand of the medicalpractitioner from the received images. The controller 310 is furtheradapted to receive the position in time of the fingertip of the medicalpractitioner from the electric field sensor 302 and determine thegesture performed by the medical practitioner from the received positionof the fingertip. The controller 310 is further adapted to generate aGUI in substantially real-time. The GUI comprises four virtual icons anda virtual representation of the hand of the medical practitioner. Theposition of the virtual representation of the hand within the GUI isdetermined using the position of the fingertip received from theelectric field sensor 302. The orientation of the virtual representationof the hand within the GUI is determined according to the determinedorientation of the hand obtained from the images received from thecamera 304.

After creating the GUI, the controller 310 transmits the GUI to thecomputer machine 312 which is in charge of displaying the medicalinformation on the display device 314. It should be understood that thecontroller 310 also transmits any detected gesture to the computermachine which retrieves the command that corresponds to the detectedgesture and executes the command. Alternatively, the controller 310 maybe adapted to determine the command that corresponds to the detectedgesture and then transmits the command to the computer machine whichexecutes the command.

In the illustrated embodiment, the computer machine 312 is adapted todisplay both the medical image 316 and the overlay GUI 318 on the samedisplay device 314. In the illustrated example, the overlay GUI 318 isdisplayed over the medical image 316 at the right bottom corner of thescreen of the display device 314. The person skilled in the art willunderstand that other configurations are possible.

As a result of the display of the medical image and the GUI on the samedisplay device, the GUI is always in the field of view 320 of themedical practitioner while he is looking at the screen of the displaydevice 314 to see the medical information displayed thereon.

In one embodiment GUI is updated in real time so that the position andorientation of the virtual hand substantially always correspond to theseof the real hand.

While the system 300 comprises a single display device 314, FIG. 20illustrates an embodiment of a system 330 that comprises two separatedisplay devices. The system 330 comprises the same camera 304, projector306, electric field sensor 302 and controller 310 as those contained inthe system 300. The system 330 further comprises a computer machine 332,a first or main display device 334 and a second or auxiliary displaydevice 336.

The computer machine 332 is adapted to display the medical image 316 onthe screen of the first display device 334 and the GUI 318 on the screenof the second display device 336. The position of the second displaydevice 336 relative to the first display device 334 is chosen so thatthe GUI 318 be contained in the visual field of view 320 of the medicalpractitioner while he is looking at the medical image 316. This can beachieved by adequately choosing the size of the screen of the seconddisplay 336 and positioning the second display device 336 adjacent tothe first display device 334. For example, the second display device 336may be in physical contact with the first display device 334. It shouldbe understood that the relative position between the first and seconddisplay devices 334 and 336 illustrated in FIG. 20 is exemplary only.

It should be understood that any adequate type of display device may beused for displaying the medical information/data and the GUI. Forexample, light-emitting diode displays, liquid crystal displays and/orthe like may be used.

While in the above description, the GUI comprises at least a virtualrepresentation of the object used by the medical practitioner tointeract with the sensing unit 202 and at least one virtual icon, itshould be understood that other configurations may be possible. Forexample, the GUI may further comprise a virtual representation of thesensing unit 202 or an element of the sensing unit 202 such as a virtualrepresentation of an electric field sensor when the sensing unit 202comprises both a camera and an electric field sensor. In anotherexample, the GUI may only comprise a virtual representation of theobject used by the medical practitioner to interact with the sensingunit 202 and a virtual representation of the sensing unit 202 or acomponent of the sensing unit such as a virtual representation of anelectric field sensor when the sensing unit 202 comprises both a cameraand an electric field sensor. In this case, the GUI comprises no virtualicons and the position of the virtual representation of the objectrelative to the virtual representation of the sensing unit is determinedaccording to the position of the object determined by the sensing unit202.

While in the above description, the sensing unit 202 is adapted todetect both the position and orientation of the object used by themedical practitioner to interact with the sensing unit 202, it should beunderstood that the sensing unit 202 may be adapted to detect only theposition of the object. In this case, the position of the virtualrepresentation of the object in the GUI is determined using the positionof the object relative to the sensing unit 202 determined by the sensingunit 202 and the orientation of the object is not represented in theGUI.

It should be understood that the GUI may correspond to a 2Drepresentation of the object and the icons and/or the sensing unit whenthe sensing unit is adapted to detect only the position of the object.When the sensing unit is adapted to determine both the position andorientation of the object relative to the sensing unit, the GUI maycomprise a 3D virtual representation of the object.

While in the above description and figures, the sensing unit isrepresented positioned on a bed, it should be understood that thisparticular position for the sensing unit is exemplary only and that thesensing unit may be positioned at any adequate position such as on atable adjacent to the bed for example. In one embodiment, the sensingunit may be a handheld device that may be hold by the medicalpractitioner and positioned on a surface such as on a bed when needed.

It should be understood that wired or wireless communication may be usedfor connecting the different elements of the above-described system.

The embodiments of the invention described above are intended to beexemplary only. The scope of the invention is therefore intended to belimited solely by the scope of the appended claims.

We claim:
 1. A system for permitting a medical practitioner to interactwith medical information, the system comprising: at least one touchlesssensor for detecting at least a 3D position of a reference object usedby the medical practitioner to touchlessly interact with the at leastone touchless sensor; and at least one control unit being incommunication with the at least one touchless sensor for: determining atouchless gesture performed by the medical practitioner using the 3Dposition of the reference object detected by the at least one touchlesssensor; identifying a command relative to the medical information thatcorresponds to the received touchless gesture and executing the commandin order to display the medical information on a screen of a displayunit; generating a graphical user interface (GUI) comprising a virtualrepresentation of the reference object, a virtual representation of theat least one touchless sensor, and a virtual representation of at leastone virtual icon, a position of the virtual representation of thereference object relative to the virtual representation of the at leastone touchless sensor within the GUI being chosen as a function of the 3Dposition of the reference object detected by the at least one touchlesssensor so as to provide the medical practitioner with a visual feedbackof a relative position between the reference object and the at least onetouchless sensor, each of the at least one virtual icon corresponding toone of a respective mode of operation, a respective user notificationand a respective system setting option; and displaying the GUI includingthe virtual representations of the reference object, the at least onesensor, and the at least one virtual icon on the screen of the displayunit along with the medical information, wherein the displayed GUIincluding the virtual representations of the reference object, the atleast one sensor, and the at least one virtual icon occupies a portionof the screen of the display unit on which the medical information isdisplayed; wherein displaying the GUI comprises displaying the virtualrepresentation of the at least one sensor at a location spaced apartfrom a sensing region of the at least one touchless sensor.
 2. Thesystem of claim 1, wherein the at least one touchless sensor is furtheradapted to detect an orientation of the reference object, an orientationof the virtual representation of the reference object within the GUIbeing chosen as a function of the orientation of the reference objectdetected by the at least one touchless sensor.
 3. The system of claim 2,wherein the at least one touchless sensor comprises a single sensoradapted to determine the 3D position and the orientation of thereference object and determine the touchless gesture performed by themedical practitioner.
 4. The system of claim 2, wherein the at least onetouchless sensor comprises a first sensor for determining the 3Dposition of the reference object and a second sensor for determining theorientation of the reference object, the touchless gesture beingdetermined by one of the first and second sensors.
 5. The system ofclaim 4, wherein the first sensor comprises an electric field sensor fordetermining the 3D position of the reference object and the secondsensor comprises an optical sensor for determining an orientation of thereference object.
 6. The system of claim 5, wherein the optical sensorcomprises a camera, the camera comprising one of a 2D camera, amonochrome camera, a stereo camera and a time-of-flight camera.
 7. Thesystem of claim 1, wherein the reference object comprises a body part ofthe medical practitioner.
 8. The system of claim 7, wherein the bodypart comprises one of a hand and at least one finger.
 9. The system ofclaim 7, wherein the reference object is made of one of a conductivematerial and a semi-conductive material.
 10. The system of claim 7,wherein the reference object comprises one of a pen, a stylus, a ball, aring, and a scalpel.
 11. The system of claim 1, wherein the commandcorresponds to a given known command from a peripheral deviceconnectable to a computer machine, the given known command correspondingto one of a mouse command, a foot pedal command, a joystick command, anda keyboard command.
 12. The system of claim 1, wherein the medicalinformation comprises a medical image, a 3D model, and any combinationor sequence thereof.
 13. The system of claim 1, wherein the commandrelative to the medical information comprises a command that causes achange of at least one characteristic of an already displayed medicalimage.
 14. The system of claim 13, wherein the at least onecharacteristic comprises at least one of a shape, a size, anorientation, a color, a brightness, text and a contrast.
 15. Acomputer-implemented method for allowing a medical practitioner tointeract with medical information, the method comprising: detecting a 3Dposition of a reference object used by the medical practitioner totouchlessly interact with at least one touchless sensor based on outputfrom at least one touchless sensor; determining a touchless gestureperformed by the medical practitioner using the detected 3D position ofthe reference object; identifying a command relative to the medicalinformation that corresponds to the received touchless gesture andexecuting the command in order to display the medical information on ascreen of a display unit; generating a graphical user interface (GUI)comprising a virtual representation of the reference object, a virtualrepresentation of the at least one touchless sensor, and a virtualrepresentation of at least one virtual icon, the position of the virtualrepresentation of the reference object relative to the virtualrepresentation of the at least one touchless sensor within the GUI beingchosen as a function of the detected 3D position of the reference objectdetected by the at least one touchless sensor so as to provide themedical practitioner with a visual feedback of a relative positionbetween the reference object and the at least one touchless sensor, eachof the at least one virtual icon corresponding to one of a respectivemode of operation, a respective user notification and a respectivesystem setting option; and displaying the GUI including the virtualrepresentations of the reference object, the at least one sensor, andthe at least one virtual icon on the screen of the display unit alongwith the medical information, wherein the displayed GUI including thevirtual representations of the reference object, the at least onesensor, and the at least one virtual icon occupies a portion of thescreen of the display unit on which the medical information isdisplayed; wherein displaying the GUI comprises displaying the virtualrepresentation of the at least one sensor at a location spaced apartfrom a sensing region of the at least one touchless sensor.
 16. Thecomputer-implemented method of claim 15, further comprising detecting anorientation of the reference object.
 17. The computer-implemented methodof claim 16, wherein said at least one touchless sensor comprises afirst sensor and a second sensor and wherein detecting the 3D positionof the reference object is performed using output from the first sensorand said detecting the orientation of the reference object is performedusing output from the second sensor, the touchless gesture beingdetermined using at least one of the first and second sensors.
 18. Thecomputer-implemented method of claim 17, wherein the first sensorcomprises an electric field sensor for determining the 3D position ofthe reference object and the second sensor comprises an optical sensorfor determining the orientation of the reference object.
 19. Thecomputer-implemented method of claim 15, wherein the reference objectcomprises a body part of the medical practitioner.